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Overview 


•  Tasks  and  role  of  naval  operators  in  traditional  system 

•  Tasks  and  role  of  operators  in  system  employing 
automated  detection  and  tracking  and  MSDF 

•  Impact  of  change  on  operator  and  operational  effectiveness 

-  based  on  research  with  other  systems 

-  based  on  observation  of  performance  with  real  systems 

•  Integration  of  DCIEM  research  with  existing  knowledge 
base 

•  Conclusions 

Defence  and  Civil  Institute  of  Environmental  Medicine  (DCIEM) 


p 

Traditional  system 

•  Sensor  Operator 

-  searches  incoming  data 
for  possible  targets 

-  detects,  tracks  and 
analyses  possible 
targets 

-  passes  on  reports  of 
possible  targets 

•  Track  Supervisor 

-  adds  new  contacts  to 
tactical  display 

-  updates  position  of 
existing  contacts 

-  correlates  contacts 
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Combat  System  Operator  O rganization 


Traditional  system  (2) 


•  Warfare  Director 

-  coordinates  tactical  picture 

-  recommends  classification 

-  provides  command  with  update  of  tactical  picture 
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Why  introduce  automation 


•  More  sensitive  sensors  and  increased  computing  power 

-  amount  of  data  increased  substantively 

•  Increase  in  number  and  type  of  sources  of  information 

•  New  concept  of  operations 

-  littoral  versus  deep  water  -  more  contacts,  more  noise 

•  Reduced  manning 

-  fewer  operators  to  interrogate  data 

•  It  is  possible 
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New  concept  ? 


•  Sensor  Operator 

-  monitors  output  of  automated  system  that  detects, 
tracks  and  does  preliminary  classification  of  contacts 

-  compares  automated  system’s  decisions  with  current 
tactical  and  environmental  picture 

-  interrogates  data  if  necessary 

-  removes  false  alarms  and  passes  on  possible  contacts 
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New  concept  ?  (2) 


•  Track  Supervisor 

-  monitors  output  of  combat  data  fusion  system  that  fuses 
output  of  sensors  with  information  from  other  sources 

-  queries  system  as  to  status  of  contacts,  sources  etc. 

•  Warfare  Director 

-  monitors  output  of  multi-sensor  data  fusion  system 

-  recommends  classification 

-  provides  command  with  update  of  tactical  picture 
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Research  on  use  of  automated  systems 


•  Humans  tend  not  to  use  automated  systems  if  they  perceive 
them  as  unreliable 

•  Even  if  they  are  perceived  as  reliable,  humans  may  not  use 
the  system  in  order  to  maintain  control 

•  If  system  is  reliable  and  used,  the  human  often  fails  to 
monitor  it  adequately 

•  Use  of  automated  systems  can  lead  to  loss  of  skills  so 
operator  unable  to  intervene 

•  Automation  errors  often  difficult  to  detect  because  of 
inadequate  feedback 
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Caveat 


•  Most  of  the  research  on  human  use  of  automation  centered 
on  cockpit  automation  and  process  control 

•  Frequently  automated  tasks  can  be  carried  out  manually 

•  Automation  introduced  to 

-  reduce  workload 

-  reduce  error  rate 

-  increase  number  of  tasks  that  a  single  operator  can 
monitor 
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Review  by  Klein  Associates  Inc. 


•  Looked  specifically  at  potential  impact  of  Multi-Sensor 
Data  Fusion  (MSDF) 

•  Reviewed  in  context  of  Naturalistic  Decision  Making 

•  Based  on: 

-  literature  discussed  previously 

-  experience  with  wide  range  of  U.S.  military  systems 

•  Focus  critical 
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Potential  problems  with  MSDF 

•  Interferes  with  application  of  expertise 

-  Reduces  evaluation  of  data  when  integration  of  sources 
automatic 

-  see  what  system  found  and  not  what  missed 

•  Interferes  with  expertise  of  team 

-  verbal  or  written  communication  shared  by  everyone 

•  Slows  rate  of  learning  of  novices 

-  less  requirement  to  deal  with  data 

-  less  capability  to  handle  task  if  automation  fails 

-  less  capability  to  detect  anomalies 
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DCIEM  research 


•  Initiated  because  of  observed  operator  dissatisfaction  with 
automated  detection  and  tracking  systems 

•  Generic  target  tracking  task 

-  later  added  automated  detection  capability 

•  Details  of  studies  reported  in  series  of  presentations  and 
papers 

•  Compare  findings  with  other  research  and  concerns  of 
Klein 
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Automated  tracking 


•  Use  of  moderately  reliable  tracker  depended  on: 

-  number  of  targets  that  had  to  be  tracked 

-  capability  of  individual  to  do  task  manually 

-  experience  with  automated  tracker 

•  Even  moderately  reliable  tracker: 

-  perceived  as  reducing  workload 

-  reduced  time  on  task 

•  As  reliability  increased: 

-  detection  of  low  strength  targets  increased 

-  detection  of  automation  errors  appeared  to  decrease 
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Automated  tracking  (2) 


•  Initial  experience  with  manual  task: 

-  did  not  result  in  better  system  performance 

-  delayed  learning  how  to  find  low  strength  targets 

•  Detection  of  automation  errors: 

-  little  evidence  that  it  was  due  to  complacency 

-  poorest  when  busy  handling  other  tasks  such  as  target 
detection 

-  suggests  due  to  relatively  low  visibility  of  those  type  of 
errors 
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Automated  detection  and  tracking 


•  Varied  detection  threshold  and  tracking  reliability 

•  Little  variation  in  performance  across  conditions 

•  Best  performance  with  low  detection  threshold  and 
moderate  reliability 

-  less  time  to  remove  false  alarms  -  more  likely  that  low 
strength  targets  added  by  ADT 

-  targets  not  updated  as  consistently  -  automation  errors 
corrected 

•  No  improvement  in  detection  of  automation  errors  under 
high  reliability  tracker 
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Automated  detection  and  tracking  (2) 


•  Provided  capability  to  adjust  detection  threshold  and 
tracking  reliability 

•  Most  users: 

-  chose  a  detection  threshold  that  minimized  false  alarms 

-  tracker  parameters  that  resulted  in  tracks  being 
consistently  updated 

•  Performance  similar  to  high  reliability,  high  threshold 
condition  in  previous  study 

-  preferred  to  let  automation  handle  standard  tracks  and 
search  for  low  strength  targets  manually 
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Conclusion 


•  Automation  does  allow  operators  to  handle  higher 
workload 

-  performance  likely  to  be  higher  than  if  tried  to  do  task 
manually  but  will  likely  miss  anomalies 

•  Most  users  make  maximum  use  of  automation  unless  seen 
as  increasing  workload 

-  Any  efforts  to  increase  operator  involvement  has  to  be 
perceived  as  beneficial  in  short  term 

-  Need  to  force  interaction  with  system  in  intelligent 
manner 
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